feat(ai): add the ai resource with an OpenAI-compatible gateway#178
feat(ai): add the ai resource with an OpenAI-compatible gateway#178ItamarZand88 wants to merge 24 commits into
Conversation
|
Too many files changed for review. ( Bypass the limit by tagging |
| @@ -0,0 +1,26 @@ | |||
| import * as alien from "@alienplatform/core" | |||
|
|
|||
| // A model-less AI resource. The customer's cloud serves the inference; the | |||
There was a problem hiding this comment.
"model-less AI resource" - confusing
'embedded gateway workload ambient identity etc -- implementation details remove from example
| const api = new alien.Worker("api") | ||
| .code({ type: "source", src: "./", toolchain: { type: "typescript" } }) | ||
| .publicEndpoint("api") | ||
| // Linking injects ALIEN_LLM_BINDING and exposes the gateway as ALIEN_AI_GATEWAY_URL. |
There was a problem hiding this comment.
no need for this comment to be here
| if (!question) { | ||
| return c.json({ error: "pass a question as ?q=..." }, 400) | ||
| } | ||
| const llm = ai("llm") |
There was a problem hiding this comment.
ai("llm") is very confusing :) lets think of better name than "llm"
| const completion = (await llm.chat.completions.create({ | ||
| model, | ||
| messages: [{ role: "user", content: question }], | ||
| })) as { choices?: Array<{ message?: { content?: string } }> } |
There was a problem hiding this comment.
this code is a bit messy.. why is this necessary? need a much cleaner API
| @@ -0,0 +1,178 @@ | |||
| //! Embedded, protocol-agnostic AI gateway: a loopback HTTP server that injects the | |||
There was a problem hiding this comment.
Let's rename this crate to alien-ai-gateway
| @@ -0,0 +1,148 @@ | |||
| //! Live verification against real AWS Bedrock. Ignored by default — it makes a | |||
| //! real inference call and needs ambient AWS credentials in the environment. | |||
There was a problem hiding this comment.
Let's not ignore these tests by default, they're very important (same for foundry vertex etc). And lets run them and make sure they're all passing.
You can load our alien-test-target account credentials from ../env.test - see this pattern in the alien-bindings tests.
| //! real inference call and needs ambient AWS credentials in the environment. | ||
| //! | ||
| //! Run it with: | ||
| //! eval "$(aws configure export-credentials --profile <p> --format env)" |
There was a problem hiding this comment.
no need for this thing once we'll load ../env.test
| @@ -0,0 +1,52 @@ | |||
| //! Node-API addon for the alien AI gateway. | |||
There was a problem hiding this comment.
Hmmm.. let's think if we really need this crate. Maybe we can just start an ./alien-ai-gateway process from the typescript package?
discuss a little bit with claude what are the advantages / disadvantages of this
| @@ -0,0 +1,77 @@ | |||
| # `@alienplatform/ai-gateway` — package layout contract | |||
| @@ -0,0 +1,57 @@ | |||
| { | |||
There was a problem hiding this comment.
Let's think about it for a second. Do we really need a separate package for this? or should this just be inside @alienplatform/bindings?
There was a problem hiding this comment.
Looked at folding this into @alienplatform/bindings and decided to keep it separate.
Main reason: bindings is in every worker (kv/storage/queue/vault), so if the gateway lives there too, every worker ships the alien-ai-gateway binary even when it never calls ai(). Keeping it on its own means only AI workers carry it.
They're also different beasts now. bindings is a napi addon loaded in-process, the gateway is a standalone binary we spawn. Different build and staging (it's why alien-build has the Napi/Binary split). One package for both felt off.
Re binary size: same conclusion. There's only one napi addon left now (bindings), and the gateway binary only gets embedded into compiled workers that actually use ai(), so a normal worker doesn't grow. Merging is what would bloat everything.
Left it separate for now, lmk if you disagree.
| // module for each. They're real runtime dependencies, so the package stays | ||
| // self-contained. | ||
| external: [ | ||
| "@alienplatform/bindings", |
There was a problem hiding this comment.
do we really need both @alienplatform/bindings and @alienplatform/ai-gateway and not put them on the same thing? is there very large impact on the binary size?
There was a problem hiding this comment.
Kept them separate (more on the package.json thread). The size question actually argues for it: the gateway binary only lands in compiled workers that call ai(), so merging into bindings would put it in every worker instead. No size win from combining.
Apply Alon's #178 review: - Replace the alien-ai-gateway-node napi addon with spawning the standalone alien-ai-gateway binary. The SDK uses ALIEN_AI_GATEWAY_URL when a container launcher set it, else spawns the binary and reads the URL it prints; compiled Workers embed the binary and extract+spawn it at runtime. Deletes the addon crate + npm prebuilds; alien-build stages the binary (AddonKind::Binary). - Type chat.completions/responses.create via OpenAI's result types (types-only devDependency), dropping the caller-side casts. - Rename crate alien-gateway -> alien-ai-gateway. - Trim the ai-quickstart example; rename the binding llm -> assistant. - Live tests load .env.test (Bedrock un-ignored; Foundry/Vertex/Bedrock-Claude gated pending minted tokens + model grants); exclude the live binaries from the fast CI job.
There was a problem hiding this comment.
Did a deeper pass on the non-example parts. Two high-severity issues (a BYO provider-validation gap, and a process-lifecycle bug that hangs one-shot callers) plus two cheap fixes (a 2 MB proxy body limit and a Frozen-vs-Live GCP gap). The gateway proxy core itself looks solid.
| function providerBaseUrl(provider: string): string { | ||
| const override = process.env.ALIEN_AI_LOCAL_BASE_URL | ||
| if (override) return override.replace(/\/$/, "") | ||
| return provider === "anthropic" ? "https://api.anthropic.com" : "https://api.openai.com" |
There was a problem hiding this comment.
🟠 High. An unvalidated provider can silently send a BYO key to the wrong host.
provider is a free-form string at every layer (ExternalAiBinding.provider: String, binding.ts z.string(), no allowlist), and providerBaseUrl here falls back to https://api.openai.com for anything that isn't exactly "anthropic". _postSurface then attaches the BYO key as Authorization: Bearer to that base. So a binding with provider: "Anthropic" (casing), "google", "groq", or a typo, and no ALIEN_AI_LOCAL_BASE_URL override, ships the customer's key to OpenAI. Only the openai BYO path is tested, so these paths are unexercised.
Suggested fix. Validate provider against a known set at the resolve boundary and fail closed instead of defaulting to OpenAI.
(Correcting my earlier note: the "anthropic" branch itself is fine. https://api.anthropic.com/v1/chat/completions with Bearer is Anthropic's OpenAI-compat endpoint. The one real gap on that path is responses.create, since the compat layer doesn't serve /v1/responses, so a BYO-anthropic Responses call would 404 while chat/completions works.)
| child.on("exit", onGone) | ||
| child.stdout?.resume() | ||
| child.stderr?.resume() | ||
| child.unref() |
There was a problem hiding this comment.
🟠 High (scoped to one-shot processes). child.unref() here does not let a short-lived host process exit.
unref() drops only the process handle, but the resume() calls just above put stdout/stderr into flowing mode, which keeps those pipe handles referenced. I reproduced this with a detached long-running child: after resume() + child.unref(), the parent was still alive until the child exited. Long-running Workers want to stay up anyway, but a CLI, batch job, or test process that calls ai() can hang forever.
Suggested fix. Unref the stdout/stderr pipe handles after draining them, or restructure stdio so no referenced pipes remain after the readiness handshake. Note that ChildProcess.stdout/stderr are typed as Readable, so direct child.stdout?.unref() does not type-check even though the runtime pipe is a net.Socket; use a type-safe socket/narrowing helper. Add a one-shot subprocess test that starts the gateway and asserts the host exits.
| client: reqwest::Client::new(), | ||
| models_cache, | ||
| }); | ||
| Router::new() |
There was a problem hiding this comment.
🟡 Should-fix (easy). Not merge-blocking, but it silently rejects valid large requests.
The handlers extract body: Bytes and the router never calls DefaultBodyLimit::disable(), so axum 0.8's default 2 MB limit 413s requests before they reach the upstream. That's reachable for vision/base64 images and long tool-heavy conversations.
Suggested fix. Since this is a pure proxy, disable the limit and let the upstream enforce its own: Router::new().layer(DefaultBodyLimit::disable()).
| //! Vertex AI endpoint without a cloud round-trip. | ||
| //! | ||
| //! The `aiplatform.googleapis.com` API enablement is handled by the | ||
| //! `GcpServiceActivationEmitter` when the preflight injects a |
There was a problem hiding this comment.
🟡 Should-fix (easy). Breaks the Frozen/Terraform GCP path only (the native controller is fine).
The doc here says aiplatform enablement is handled by GcpServiceActivationEmitter via an injected ServiceActivation, but get_required_services() in gcp_service_activation.rs has no "ai" arm, so nothing injects it. terraform apply still succeeds (the emitter just produces no google_project_service); the failure surfaces later at gateway runtime as SERVICE_DISABLED when Vertex is first called on a project that didn't already have the API on. The native controller enables it at runtime (ai/gcp.rs), so only Terraform-mode GCP deploys are affected.
Suggested fix. Add an "ai" arm to get_required_services().
| // A small curated chat-model list for the BYO-key picker (we don't proxy the provider's own | ||
| // /v1/models, which returns hundreds of non-chat entries). | ||
| function defaultModels(provider: string): string[] { | ||
| return provider === "anthropic" |
There was a problem hiding this comment.
🟠 High (BYO Anthropic). getAvailableModels() returns only two retired model families for this provider.
claude-3-5-sonnet-latest and claude-3-5-haiku-latest point at Claude 3.5 families whose pinned models were retired on 2025-10-28 and 2026-02-19 respectively. Anthropic documents that requests to retired models fail: https://platform.claude.com/docs/en/about-claude/model-deprecations. Because staticModels bypasses the provider models endpoint, a BYO-Anthropic picker receives only unusable IDs and the first real completion fails.
Suggested fix. Return active IDs such as claude-sonnet-5 / claude-haiku-4-5 and cover the Anthropic branch with a request test. Better, query the provider models endpoint so this list does not silently age out again.
| /// Cache-then-fetch the workload's projected-identity token from the instance metadata | ||
| /// service. `source` must be `Gcp` or `Azure`. | ||
| async fn metadata_token(&self, source: &BearerSource) -> Result<String> { | ||
| if let Some((tok, exp)) = self.cache.lock().await.as_ref() { |
There was a problem hiding this comment.
🟠 High on projected-identity GCP/Azure workloads. This cache miss is not single-flight.
The mutex protects only the lookup and is released before the metadata request. The first /v1/models call uses join_all to launch every model probe concurrently (14 on GCP, 12 on Azure); each probe authorizes independently, so they can all observe an empty cache and hit IMDS at once. The same stampede happens when a token expires under concurrent inference. If IMDS throttles, model probes become indeterminate, the list is deliberately left uncached, and the next call repeats the fan-out.
Suggested fix. Serialize refreshes behind the cache lock or store a shared in-flight refresh future/OnceCell, then add a concurrent cold-cache test that asserts exactly one metadata request.
| @@ -0,0 +1,2137 @@ | |||
| //! The pure proxy: route a request to the model's native cloud endpoint, inject the | |||
There was a problem hiding this comment.
🟡 Structural. This new module lands at 2,137 lines and has too many reasons to change.
The first ~974 lines combine shared routing/transport, cloud URL selection, three provider adapters, Bedrock request normalization, and an event-stream decoder; another ~1,160 lines of tests live in the same file. availability.rs already imports several of these internals, so the boundary is starting coupled rather than cohesive.
Suggested fix. Decompose before this becomes the permanent extension point: keep dispatch/common transport in router/mod.rs, move Bedrock, Vertex, and Foundry handling into focused modules, and isolate the Bedrock event-stream codec with its tests. That leaves future provider changes local and keeps each module reviewable.
Introduce built-in AI: an `ai` resource that provisions keyless, in-account model access through an embedded gateway. Adds the Rust gateway engine and napi addon, the TypeScript SDK wrapper, the per-cloud controllers and setup emitters (AWS Bedrock, GCP Vertex, Azure Foundry), the ai/invoke permission sets, and quickstart examples.
- The napi error envelope now carries httpStatusCode and hint, so a gateway startup failure (e.g. BindingConfigInvalid at http_status_code 400) is no longer flattened to a 500 when it crosses into JS. - parseSse preserves the malformed-chunk cause as a structured error source instead of folding it into the message string. - Name the condition the anthropic-beta merge test enforces rather than the old bug it guards against.
The ai() client resolved its napi addon by filesystem lookup, which cannot work inside a `bun build --compile` single-file binary — so a compiled Worker calling ai() failed while kv/storage/queue/vault worked. Mirror the bindings embedded-addon mechanism end to end: - The ai-gateway loader gains an embedded slot (registerEmbeddedAddon); loadAddon prefers it over the filesystem/prebuild resolution that can't run in a binary. - ai-gateway/native installEmbeddedAddon() registers the bun-embedded addon. - @alienplatform/sdk/native installs both the bindings and ai-gateway addons; the SDK keeps both external so the compiled binary shares one module for each. - alien-build stages the ai-gateway addon alongside bindings, best-effort: a Worker that resolves ai-gateway transitively through the SDK but never calls ai() still builds (a missing addon for the target is skipped, not an error). Verified by the package-layout compile-smoke (a compiled binary resolves the SDK's ai + storage after both staged .node files are removed) and a loader unit test that loadAddon prefers the embedded addon.
The Postgres runtime binding (packages/bindings/src/postgres.ts, getPostgresConnection, cloud secret-manager deps) and the AI+Postgres chatbot example rode in from the shared source branch. main already ships the Postgres resource and dials the DB directly, and PACKAGE_LAYOUT.md keeps getPostgresConnection off the SDK root, so this copy is superseded. Remove it so this PR stays AI-only.
Apply Alon's #178 review: - Replace the alien-ai-gateway-node napi addon with spawning the standalone alien-ai-gateway binary. The SDK uses ALIEN_AI_GATEWAY_URL when a container launcher set it, else spawns the binary and reads the URL it prints; compiled Workers embed the binary and extract+spawn it at runtime. Deletes the addon crate + npm prebuilds; alien-build stages the binary (AddonKind::Binary). - Type chat.completions/responses.create via OpenAI's result types (types-only devDependency), dropping the caller-side casts. - Rename crate alien-gateway -> alien-ai-gateway. - Trim the ai-quickstart example; rename the binding llm -> assistant. - Live tests load .env.test (Bedrock un-ignored; Foundry/Vertex/Bedrock-Claude gated pending minted tokens + model grants); exclude the live binaries from the fast CI job.
getAvailableModels now returns only the models the deployment's cloud has actually enabled, probed at runtime, instead of the static catalog superset. The gateway sends a tiny max_tokens:1 request per catalog model on the first /v1/models (cached per process), signed with the workload's ambient credential, and keeps a model only if it authenticated (2xx/429/400); 401/403/404 drop it. No permission change: the probe rides the existing ai/invoke inference grant. Enriches AiModel with provider + displayName for a picker, adds a static activation field documenting each model's one-time enablement step, iterates the e2e over every listed model, and updates the example to discover-then-pick.
…uilds The release pipeline built the deleted alien-ai-gateway-node napi crate. It now builds the alien-ai-gateway binary in the addon matrix (darwin legs on the addon's targets; linux legs statically against musl, so one binary per arch serves both glibc and Alpine), generates the six per-triple prebuild manifests at publish time instead of keeping them checked in, and publishes them before the wrapper with exact-version optionalDependencies. The dry-run path covers the whole build and publish flow, matching publish-bindings.
…ration test /v1/models now probes every catalog model against the mock upstream, and an unmatched probe 404s and drops the model, so the Claude catalog assertion needs the InvokeModel path answered.
The deployment reconcile polls a deep async chain on a single stack: the executor drives each resource controller, which drives the cloud SDKs' tower/hyper/rustls futures. In debug builds those frames are un-inlined and the chain needs about 3MB, over tokio's 2MB default thread stack, so the alien-test cloud e2e (push_/pull_) overflows its libtest test thread and aborts during initial setup. Release binaries are unaffected (inlined frames fit the 2MB default, confirmed by running the same deploy in release), so this only sizes dev/test threads via a cargo [env] entry. 8MB matches the headroom alien-worker-runtime already gives its runtime for the same deep-async reason.
The availability probe counted 400 as "enabled", assuming it proved the endpoint authed and only rejected the minimal probe body. The cloud e2e disproved that: Bedrock answers "The provided model identifier is invalid" with 400 for a model the account cannot address, so qwen3-coder-next and claude-mythos-5 were listed by getAvailableModels and then failed on the first real call. The probe body is minimal and well formed, so a 400 is about the model rather than the request. Classifying it as unavailable keeps the contract the e2e asserts: every listed model is invocable. A model that only ever answers 400 now drops out of the list instead of being advertised and breaking at runtime.
Draining the child's stdio with resume() keeps the pipe sockets referenced, so a process that calls ai() once and returns never exits. Unref both stdio sockets alongside the child.
The native controller enables aiplatform.googleapis.com at runtime, but a Frozen/Terraform GCP deploy relies on this activation being injected here, or the first Vertex call fails with SERVICE_DISABLED.
Hold the metadata cache lock across the IMDS fetch so a burst of concurrent probes (the /v1/models fan-out authorizes every model at once) collapses to one refresh instead of stampeding the metadata service.
providerBaseUrl defaulted any provider other than "anthropic" to OpenAI's endpoint, and the projected BYO key is attached as a bearer token there, so a typo or unlisted provider shipped the customer's key to OpenAI. Resolve only the providers we have a base URL for and throw otherwise; the ALIEN_AI_LOCAL_BASE_URL override still allows any OpenAI-compatible provider.
The BYO-Anthropic model picker returned retired claude-3-5-* ids. Return the current-generation aliases api.anthropic.com serves instead.
router.rs had grown to hold the transport, per-cloud URL dispatch, three provider adapters, the Bedrock body normalization, and the event-stream codec in one file. Move each provider's request path into its own module under router/ (bedrock/vertex/foundry) plus the event-stream decoder (eventstream), keeping mod.rs as the transport and dispatch. Behavior-preserving; the shared integration test harness stays in mod.rs.
The gateway is a pure proxy, so the upstream enforces its own body size, but axum's 2 MB default rejected large legitimate requests (base64 vision images, long tool-heavy conversations) with 413 before they ever left us. Disable the limit on the router.
reqwest has no default request timeout, so a hung instance metadata service would wedge /v1/models forever — and holding the single-flight lock across the fetch makes that block every waiting caller, not just one. Cap the fetch at 10s (the endpoint is link-local and answers in well under a second) so a real hang fails fast instead of hanging.
Anthropic's OpenAI-compatible host serves /v1/chat/completions but not the Responses API, so a BYO-Anthropic responses.create() returned an opaque 404. Detect the Anthropic native base and throw a clear reason before the request; an ALIEN_AI_LOCAL_BASE_URL override resolves elsewhere and is not blocked.
The three new AI ImportData types were never registered in the snapshot map, so the pinned schemas had no AI entry and the test passed without covering them.
Claude over classic InvokeModel signs with bedrock, not bedrock-mantle; bedrock-mantle serves the Responses API. The doc had the two the other way round.
4176efd to
fe0ac8f
Compare
3 participants
Summary
Adds an
airesource that gives a workload an OpenAI-compatible endpoint for model inference, in two modes. By default it's keyless: for the host cloud's own models — Bedrock on AWS, Vertex on GCP, Foundry on Azure — a loopback gateway signs each request with the workload's ambient cloud identity, so there are no model API keys to store. A workload can instead bring its own provider key (e.g. OpenAI), in which case the client talks to that provider directly with the vault-resolved key and the gateway isn't involved.What happens on the keyless path, when a workload declares an
aibinding for one of the host cloud's models:InvokeModel, VertexrawPredict, or the Foundry Anthropic endpoint — and signs it with the workload's ambient cloud credentials. ← the heart of the changeFor a cloud's own models, this changes AI access from managing model API keys to using the cloud account the workload already runs in.
What I did
airesource to the SDK, plus the controllers and per-cloud setup that grant a workload access to that cloud's models.aibinding can point at an external provider with the workload's own key (vault-resolved, redacted in logs), which bypasses the gateway and calls the provider directly.getAvailableModelsreport real availability: the gateway probes each catalog model with a one-token request under the workload's own inference credential and lists only what the cloud actually has enabled (a 429 counts as enabled but rate-limited; 400/401/403/404 drop the model). Each entry carriesprovideranddisplayNameso an app can build a model picker, and the example README documents each model's one-time enablement step per cloud.chat.completions.createandresponses.createreturn OpenAI's own result types (a types-only dev dependency), so callers read.choices[0].message.contentwith no cast.ai-quickstart-tsexample built around "list the available models, then pick one".Files touched
crates/alien-ai-gateway— the gateway engine (router, model catalog, availability probe, per-cloud translation + signing) and thealien-ai-gatewaylauncher binary.packages/ai-gateway— the TypeScript wrapper: resolves and spawns the gateway binary, and the ambient vs BYO-key connection resolution.crates/alien-core,packages/core,packages/sdk— theairesource type, the model catalog with per-model activation metadata, and the SDK surface (plus embedding the gateway binary into compiled Workers viapackages/package-layoutandcrates/alien-build).crates/alien-infra— the AI controllers and runtime wiring.crates/alien-terraform+crates/alien-cloudformation— the per-cloud setup emitters and their snapshots.crates/alien-permissions/permission-sets/ai/*— theai/invokegrant (plus heartbeat/management/provision).examples/ai-quickstart-ts— a runnable example.How I tested
/v1/models: 36 of the 42 AWS catalog models came back listed and enriched, and the 6 dropped were Claude ids that account/region cannot serve.bun build --compileof a Worker that callsai()— the embedded gateway binary is extracted and spawned from inside the single-file binary and serves a loopback URL (the package-layout compile smoke).terraform validateon the AI setup-emitter output and the CloudFormation snapshot suite; generated-schema drift check clean. The live Bedrock test reads the shared test env and runs in the credentialed job.getAvailableModelsreturns and invokes each one, failing if a listed model is not invocable. That run passed after it caught a real filter bug: Bedrock answers400 "The provided model identifier is invalid"for a model the account cannot address, and I had been counting 400 as available, so it now drops the model (400 joins 401/403/404) and the list only ever contains invocable models.terraform validate.One non-AI fix rides along
The cloud e2e for this PR overflowed its test thread's stack during deploy. It is a pre-existing issue, independent of the AI code: the deployment reconcile polls a deep async chain (executor to controllers to the cloud SDKs' tower/hyper/rustls futures) that needs about 3MB in a debug build, over tokio's 2MB default thread stack. Release binaries fit 2MB and are unaffected.
RUST_MIN_STACK=8MBfor dev/test threads via.cargo/config.tomlfixes it, matching the headroomalien-worker-runtimealready gives its runtime. It rides along here because this PR's own e2e could not run without it.I also ran a security review on the diff. What it checked:
ai/invokegrants inference only — AWSbedrock:InvokeModel*(+ mantleCreateInferencescoped toproject/default, notproject/*); a GCP custom role of justaiplatform.endpoints.predict/explain(notroles/aiplatform.user, which carries deploy + dataset-export); Azure "Cognitive Services OpenAI User" (data-plane only). No deployment writes or data reads.crates/alien-permissions/permission-sets/ai/invoke.jsonc./v1/modelsprobe sends a one-token inference request signed with that same inference-only grant; it adds no control-plane permission, cannot enable or subscribe anything, and its logs carry only the model id and status, never request or response bodies.crates/alien-ai-gateway/src/availability.rs.Debugimpl redacts it (crates/alien-core/src/bindings/ai.rs) and it's resolved into the worker environment, not logged or held in control-plane state.crates/alien-ai-gateway/src/config.rs.crates/alien-ai-gateway/src/lib.rs.crates/alien-ai-gateway/src/config.rs.Nothing turned up.